Writing MatlabOpaque objects to MAT-files #208

foreverallama · 2025-11-21T06:29:59Z

Part of a series of commits to support write operations for MatlabOpaque objects. For now I just rearranged the methods to make it easy for me to update the code.

I'll first add to MAT_subsys.jl before updating MAT_HDF and MAT_v5.

foreverallama · 2025-11-24T14:13:19Z

CC @matthijscox

Added some methods to write subsystem data... WIP. Basically just translating Python code into Julia here, so would need a review.

I had to make some small adjustments to incorporate the write logic. I updated some metadata fields from Vector{UInt32} to Vector{Vector{UInt32}}. The main reason is that when serializing nested objects, full metadata for the outermost object is saved before saving metadata for any inner nested objects. The length of the metadata fields are variable, so instead of using a position-based update logic (where you might need to recalculate positions every time), I went with a mutable vector insertion instead. This results in the Vector{Vector{UInt32}} type. Since during load, we're only expecting one vector, I simply updated the loading logic to use metadata_field[1][idx] instead of metadata_field[idx].

I don't think performance would take a hit when splatting these finally as we don't expect too many objects in a MAT-file (after splat the total FileWrapper metadata length could be around thousands at best for really complex class definitions). There might be a better way to code the whole logic but until then this should work.

I also added a new object cache for saving that would key by id(MatlabOpaque), and renamed the existing one to load_object_cache.

A side note, I was able to figure out what the _c2 metadata field contains. It's something called a class alias. Accordingly I renamed the field mcos_class_alias_metadata. Essentially, this allows you to create an object instance using an alias. The _c2 metadata field contains the latest alias name, which is also saved in subsystem.mcos_names. However, the object itself is saved by the oldest alias which would be stored in the MatlabOpaque.class attribute. I don't think its super important to support this feature here, but we could consider adding a new attribute alias to MatlabOpaque if required.

matthijscox · 2025-11-25T08:11:26Z

Yeah this whole class system is beyond me :)
I don't see any obvious issues with the code on first glance.

I do notice there are zero tests for the new code. I guess you cannot create a minimal write test yet?
I also don't know if we want to test any of the subsystem internals.

foreverallama · 2025-11-25T08:47:04Z

Code's not complete yet,. Still working on it, trying to squeeze in a little when I'm free. No point in testing internals as the write tests would take care of it anyways

…stem

foreverallama · 2025-11-25T17:26:45Z

Most of the code's ready. It's not working yet I'll debug soon and in the meantime add some sample tests

foreverallama · 2025-11-26T11:10:24Z

src/MAT_subsys.jl

+    push!(fwrap_data, Any[])
+    append!(fwrap_data, subsys.prop_vals_saved)
+
+    empty_struct = MatlabStructArray([]) #! FIXME


@matthijscox can you help me create a 1x0 empty struct here? I'm struggling with the syntax a bit.

The fields _c3 and prop_vals_defaults should contain a Nx1 cell array, and each cell contains a 1x0 empty struct with no fields. Here N = no. of unique classes in the file plus one.

On my phone right now, but can you try something like MatlabStructArray(String[], (1,0))

Reading the code, this might not yet be possible. I always estimate the dimensions from the first column, but if there are no columns... Might be we have to explicitly add the dimensions to the MatlabStructArray type, so that it also works without any columns.

Does the empty struct array have any special features inside the HDF5 format?

So far I've been able to perform RW operations between MAT.jl and matio with the current code, but unfortunately MATLAB doesn't seem to be able to load files saved by MAT.jl.

It's going to take a while to pinpoint the issue as there are a few differences in the final HDF5 file structure between MAT.jl and MATLAB. At this point I cannot say if its because of the issue with empty structs. Let the MatlabStructArray type be as is for now, until I can identify the problem

After some experimenting, I believe the issue is because of the struct field names. It looks like if the field name attribute exists, then MATLAB tries to access elements of that field during load, regardless of the struct being empty itself (this is only for object deserialization in subsystem where some kind of assignment operations are happening in the background, not regular struct loading)

I don't think there's any use case for empty structs from a user perspective, so there's no need to update the definition of MatlabStructArray itself. Instead, we can use some other kind of marker that we can detect and specifically handle the empty struct case with no fields. Can you push a fix for this?

Also, we might need to support something similar for MatlabOpaque type with empty dimensions like 1x0 0x0 or 0x1. Is this possible currently? I've seen these kinds of objects in .fig files IIRC. It's not required for normal use, but MATLAB uses these placeholders in special types.

Do you want me to define a special EmptyStructArray type? Or also to create the write function for it? I don't know the HDF5 specs for such a type... is there an easy way to load an example fully empty (no columns, 1x0 matrices) struct array type?

easy way to load an example fully empty (no columns, 1x0 matrices) struct array type

Sorry, yeah that's a good point. I'll check this out first

* subsystem shared metadata fields use these 0x0 structs

foreverallama · 2025-11-28T08:54:08Z

src/MAT_HDF5.jl


 # Write a struct from arrays of keys and values
 function m_write(mfile::MatlabHDF5File, parent::HDF5Parent, name::String, k::Vector{String}, v::Vector)
+    if length(k) == 0


I added the ability to write empty dicts as 0x0 structs over here. The subsystem data works with 0x0 structs as well in MATLAB.

This eliminates the need for any new struct markers for now.

Alright. Are 0x0 structs also read as empty dicts?

Yup, all of 1x0, 0x1 and 0x0 are read as empty dicts already. But I hardcoded 0x0 for empty dict during write.

So this might go against the point raised in #214 where an empty dict is being written as a 1x1 struct with no fields. Instead I guess we could either:

Write empty dicts as 1x1 structs with no fields. For 1x0, 0x0 dims we could maybe use a special key __dims__ that contains a tuple with the dimensions. I think its very unlikely to require reading 1x0, 0x0 structs (without fields), we just need write functionality

Create a new marker for empty structs with no fields

Update MatlabStructArray to handle empty structs

What do you suggest?

It's a rather special corner case. I'm okay with a new type that we keep internal for the writing, and then we can always consider to refactor later:

struct EmptyStruct dims::Vector{Int} end

foreverallama · 2025-11-28T08:59:38Z

It appears to be working now. I tried a minimal test below and it can be read in all of MAT.jl, matio and MATLAB.

sample_obj = MatlabOpaque(Dict(
    "a" => reshape([10.0], 1, 1),
    "b" => reshape([1.0, 2.0, 3.0], 3, 1)
), "TestClass")

sample_dict = Dict(
    "var" => sample_obj,
)

MAT.matwrite("test.mat", sample_dict)

I'll now focus on some simple tests covering user-defined objects and handle class objects. I still need to add support for object arrays. If I've missed anything else let me know

foreverallama · 2025-11-29T14:23:22Z

Added support for writing object arrays
Added simple test for basic objects, handle class objects, object arrays and nested objects
Empty Dict is now written in as a 1x1 struct with no fields (fixes MAT.jl v0.11.1 cannot read a file it wrote, containing empty dictionary #214 )
New type EmptyStruct to write special zero-dimensional structs (for internal use)

The objects being written are loading successfully in MATLAB (2023b). With this I think the base for writing objects is covered. We can now write methods for converting from say datetime etc. to MatlabOpaque. A point to note is that subsystem data is being written in version 4. We can also write in v2 instead, though I think v4 has been the standard for a while now.

There's a special case of zero-dimensional objects, but again its an edge case that can be tackled if required.

I also opened #213 to fix #212 , it would be easier to merge that first and then fix conflicts here.

matthijscox · 2025-11-30T08:27:06Z

Do the tests now also cover the empty dict read/write case? If not, it would be good to add a test for that, so we don't accidentally break it again in the future.

foreverallama · 2025-11-30T09:48:55Z

Good point. I added a write test for an empty dict. It is loaded back as an empty dict, which should be a 1x1 struct with no fields in MATLAB.

matthijscox · 2025-11-30T10:47:40Z

test/write.jl

+        @test read_var["var1"] === read_var["var2"]  # same object
+    end
+end
+


I don't see tests for the special FileWrapper cases? Is that intentional?

Which one do you mean?

matthijscox · 2025-11-30T15:01:36Z

src/MAT_HDF5.jl

        write_dataset(cset, ctype, refs)
-        write_attribute(cset, name_type_attr_matlab, "cell")
+        if object_decode == UInt32(3)
+            write_attribute(cset, object_decode_attr_matlab, object_decode)


is this (FileWrapper) case tested?

can this code even be reached by users?

Part 1 addressed in the other comment. As to your second question, its not reachable by users. The FileWrapper object is some kind of special container used by MATLAB which stores data in a Cell array. This is just a simple workaround to write a cell array for MatlabOpaque types whose data is a cell array.

As far as I know, no other MatlabOpaque types use cell arrays to store (meta)data, so it will only execute for FileWrapper.

matthijscox · 2025-11-30T15:02:53Z

src/MAT_HDF5.jl

 function m_write(mfile::MatlabHDF5File, parent::HDF5Parent, name::String, obj::MatlabOpaque)
-    error("writing of MatlabOpaque types is not yet supported")
+    if obj.class == "FileWrapper__"
+        m_write(mfile, parent, name, obj["__filewrapper__"], UInt32(3))


is this FileWrapper case tested?

It's implicitly tested with the object tests. All object data + metadata is stored within this FileWrapper__ object in a MAT-file, so it is not possible to load an object without having the FileWrapper object written correctly to a MAT-file.

matthijscox · 2025-12-01T17:57:58Z

Alright, everything looks okay to me. Do you have the power to merge already (since you are a contributor now) or do I have to make you an official maintainer for that?

foreverallama · 2025-12-02T06:43:13Z

I can't merge yet

matthijscox · 2025-12-02T14:50:56Z

I'm only a member of JuliaIO I see, so cannot add you as maintainer unfortunately. I'll just have to merge PRs I suppose

foreverallama · 2025-12-02T17:03:56Z

No issues, you can also close #214

foreverallama added 2 commits November 21, 2025 11:56

Rearrange methods for readability

dc7e37f

Add some subsys methods to save objects

4026a63

Full subsystem code, some updates to writers to support writing subsy…

dd1c995

…stem

Some fixes to writing filewrapper metadata

32c678a

foreverallama commented Nov 26, 2025

View reviewed changes

foreverallama added 4 commits November 27, 2025 18:03

Fix dimensions of data in filewrapper

8416fc2

Write nested objects in subsystem

7b846dc

Improve MAT_subsys readability

3c85b70

Fix: Write empty dicts as 0x0 structs in HDF5

46c56aa

* subsystem shared metadata fields use these 0x0 structs

foreverallama commented Nov 28, 2025

View reviewed changes

foreverallama mentioned this pull request Nov 28, 2025

MAT.jl v0.11.1 cannot read a file it wrote, containing empty dictionary #214

Open

Update empty struct handling; Add tests

1d9e8bb

Add test for empty struct 1x1

74ca2bc

matthijscox reviewed Nov 30, 2025

View reviewed changes

matthijscox merged commit 457c7d3 into JuliaIO:master Dec 2, 2025
9 checks passed

Writing MatlabOpaque objects to MAT-files #208

Writing MatlabOpaque objects to MAT-files #208

Uh oh!

Conversation

foreverallama commented Nov 21, 2025

Uh oh!

foreverallama commented Nov 24, 2025

Uh oh!

matthijscox commented Nov 25, 2025

Uh oh!

foreverallama commented Nov 25, 2025

Uh oh!

foreverallama commented Nov 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

foreverallama commented Nov 28, 2025

Uh oh!

foreverallama commented Nov 29, 2025

Uh oh!

matthijscox commented Nov 30, 2025

Uh oh!

foreverallama commented Nov 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matthijscox commented Dec 1, 2025

Uh oh!

foreverallama commented Dec 2, 2025

Uh oh!

Uh oh!

matthijscox commented Dec 2, 2025

Uh oh!

foreverallama commented Dec 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants